Posted 25 January 2011
"All in all, the BOM letter offers little or no assurance that NIWA scientists have got it right this time. At most, it offers tepid support to NIWA’s Review Report." Coalition's comment on NIWA claims of Australian Bureau of Meteorology review of adjustments of New Zealand seven-station temperature records.
TEPID SUPPORT FOR NIWA BY AUSTRALIAN BUREAU OF
METEOROLOGY
The Australian Bureau of Meteorologists (BOM), like the New Zealand Institute of Water and Atmospheric Science (NIWA), is an advisory group of Government scientists responsible for the compilation and maintenance of official temperature records.
Some are critical of the selection of BOM to review work by NIWA, as both groups have been widely criticised (especially in the blogosphere) for applying the same biases and questionable adjustment methods. See, for example, http://joannenova.com.au/2010/09/australian-temperatures-in-cities-adjusted-up-by-70.
As climate archivists, both agencies are extensively engaged in the work of the IPCC; and both are firmly of the school of thought led by Professor Phil Jones of the East Anglia CRU.
The NZ Climate Science Coalition wrote to Science Minister Wayne Mapp, suggesting the appointment of two genuinely independent reviewers, and putting forward names of highly regarded scientists and statisticians. The letter pointed out:
“Appointment of two additional reviewers need not create any delays or distractions. Irreconcilable differences are unlikely, once all the data and metadata are disclosed, but, if they do arise, it is far better that the differing views are transparent and defined. The reviewers' role is only advisory, and NIWA will retain full control of both the process and outcomes, which must, however, include the publication in full of the reviewers’ report.”
This suggestion was not taken up.
Minister Mapp advised Parliament last February that the NZ Temperature Record was to be reviewed by Brett Mullan, and then peer-reviewed internally by David Wratt and James Renwick and externally by BOM. The work would then be submitted “as a paper to a scientific journal where it would be subject to the normal independent peer-review process”.
A “normal independent peer-review” sought by a journal editor is reasonably well established by convention. In the best instances, the data and metadata are also made available so that the scientific or statistical work is reproducible by interested outside experts. But what are non-independent “internal and external peer-reviews”? They are not defined, and may mean whatever the user intends them to mean.
An analogous guide is provided by reviews of historical financial information, as laid down by International Standard ISAE 3000. Clearly, a peer-review does not reach the stringency of an “Audit” where the reviewer certifies that the record presents a “true and fair view”. ISAE 3000 requires the terms of any “Assurance Engagement” to be recorded in writing, defining such issues as evidence sources and materiality levels, and demands “an attitude of professional scepticism”.
The engagement letter between NIWA and BOM remains confidential. But the role actually carried out by BOM appears to be the ISAE 3000 equivalent of a “Limited Assurance” – the lowest standard that a member of the NZ Institute of Chartered Accountants is ethically permitted to undertake.
BOM makes clear that its review is not a re-analysis of the record and is constrained to the “level of information supplied”. BOM’s formal assurance is stated in a single sentence:
“In general, the evidence provided by NIWA supports the homogeneity corrections that have been applied to the temperature record to create the ‘seven station’ series.”
This can be interpreted as saying: The material which NIWA chose to give us was supportive of their conclusions – or – NIWA offered us no evidence contradicting their own opinions.
But this needs to be read in the context of the many express caveats. NIWA’s data and methodology are assumed to be “an accurate representation of the actual analysis undertaken”. BOM was “not in a position to question all of the underlying analyses and data that have contributed to the final results”. Above all, there was no attempt to “independently determine the sensitivity of ... the analysis methodology.”
The final sentence of the BOM letter has a less constrained assurance:
“It is also clear that a number of significant adjustments (as identified by NIWA in the reports) are clearly required for the raw/composite station series owing to inhomogeneities which would otherwise artificially bias results”
But this is not contentious. Many inhomogeneities certainly exist, but the question is whether the NIWA adjustments have made the record better or worse? And with what degree of certainty?
BOM reveals that it did not have “full access to the raw and modified temperature data and metadata”. This is both contrary to assurances offered by Minister Mapp and the express injunction stated by the UK House of Commons in its recent report on “Climategate:
"Climate science is a matter of great importance and the quality of the science should be irreproachable. We therefore consider that climate scientists should take steps to make available all the data that support their work (including raw data) and full methodological workings (including the computer codes). Had both been available, many of the problems at UEA [University of East Anglia] could have been avoided".
All in all, the BOM letter offers little or no assurance that NIWA scientists have got it right this time. At most, it offers tepid support to NIWA’s Review Report.